Hive: Push Iceberg table property values to HMS table properties #2123
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pvary
reviewed
Jan 20, 2021
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
Outdated
Show resolved
Hide resolved
pvary
reviewed
Jan 20, 2021
mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
Outdated
Show resolved
Hide resolved
pvary
reviewed
Jan 20, 2021
mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
Outdated
Show resolved
Hide resolved
Collaborator
Author
|
Build failure is due to the flaky Spark tests: |
pvary
reviewed
Jan 21, 2021
hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java
Show resolved
Hide resolved
pvary
approved these changes
Jan 21, 2021
marton-bod
force-pushed
the
config_sync
branch
from
25e457a to
6d630dc
Compare
Contributor
|
What do you think about changing the title @marton-bod? I think the title below describes the change better: Thanks, |
Collaborator
Author
|
Overall I agree, I think that describes the crux of the change better. I went with the original title because we also relax the initial set of properties we push down from HMS to Iceberg in this patch. However, until we have a pre/postAlter hook on the HiveMetaHook, this is mostly a one-way street indeed. |
marton-bod
changed the title
marton-bod
force-pushed
the
config_sync
branch
from
9eb76fb to
e51d971
Compare
Contributor
|
Thanks @marton-bod for the change! |
autumnust
added a commit
to autumnust/iceberg-1
that referenced
this pull request
Feb 1, 2022
Iceberg table properties are the canonical source of truth HMS table properties should be maintained as much as possible to be in sync with the Iceberg table, but it can only happen on a best effort basis This PR makes the following changes: Ensures that all Iceberg table properties are propagated to the HMS table during HiveTableOperations commit All HMS table properties are pushed down to Iceberg as well during table creation (except for metadata location and spec props) Refactors the various property check assertions scattered throughout various test cases into a single property-focused unit test case What is left out and should be done in the future: Push property changes occurring via Hive DDL (ALTER TABLE SET TBLPROPERTIES) down to Iceberg as well. Currently this can't be done reliably because the HiveMetaHook interface only contains a preAlterTable method, but no commitAlterTable method. We'll need to extend this interface and include the change in an upcoming Hive upstream release. Author: Marton Bod <[email protected]> PR: apache/iceberg#2123 Backport Reason: To accomdate(I) for fix apache/iceberg#2328
autumnust
added a commit
to autumnust/iceberg-1
that referenced
this pull request
Feb 3, 2022
Iceberg table properties are the canonical source of truth HMS table properties should be maintained as much as possible to be in sync with the Iceberg table, but it can only happen on a best effort basis This PR makes the following changes: Ensures that all Iceberg table properties are propagated to the HMS table during HiveTableOperations commit All HMS table properties are pushed down to Iceberg as well during table creation (except for metadata location and spec props) Refactors the various property check assertions scattered throughout various test cases into a single property-focused unit test case What is left out and should be done in the future: Push property changes occurring via Hive DDL (ALTER TABLE SET TBLPROPERTIES) down to Iceberg as well. Currently this can't be done reliably because the HiveMetaHook interface only contains a preAlterTable method, but no commitAlterTable method. We'll need to extend this interface and include the change in an upcoming Hive upstream release. Author: Marton Bod <[email protected]> PR: apache/iceberg#2123 Backport Reason: To accomdate(I) for fix apache/iceberg#2328
autumnust
added a commit
to linkedin/iceberg
that referenced
this pull request
Feb 8, 2022
Iceberg table properties are the canonical source of truth HMS table properties should be maintained as much as possible to be in sync with the Iceberg table, but it can only happen on a best effort basis This PR makes the following changes: Ensures that all Iceberg table properties are propagated to the HMS table during HiveTableOperations commit All HMS table properties are pushed down to Iceberg as well during table creation (except for metadata location and spec props) Refactors the various property check assertions scattered throughout various test cases into a single property-focused unit test case What is left out and should be done in the future: Push property changes occurring via Hive DDL (ALTER TABLE SET TBLPROPERTIES) down to Iceberg as well. Currently this can't be done reliably because the HiveMetaHook interface only contains a preAlterTable method, but no commitAlterTable method. We'll need to extend this interface and include the change in an upcoming Hive upstream release. Author: Marton Bod <[email protected]> PR: apache/iceberg#2123 Backport Reason: To accomdate(I) for fix apache/iceberg#2328
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As agreed with the community:
This PR makes the following changes:
HiveTableOperationscommitWhat is left out and should be done in the future:
ALTER TABLE SET TBLPROPERTIES) down to Iceberg as well. Currently this can't be done reliably because theHiveMetaHookinterface only contains apreAlterTablemethod, but nocommitAlterTablemethod. We'll need to extend this interface and include the change in an upcoming Hive upstream release.